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Abstract. Brain shift, i. e. the change in configuration of the brain after 
opening the dura mater, is a key problem in neuronavigation. We present 
an approach to co-register intra-operative microscope images with pre- 
operative MRI data to adapt and optimize intra-operative neuronavi- 
gation. The tools are a robust classification of sulci on MRI extracted 
cortical surfaces, guided user marking of most prominent sulci on a mi- 
croscope image, and the actual variational registration method with a 
fidelity energy for 3D deformations of the cortical surface combined with 
a higher order, linear elastica type prior energy. Furthermore, the actual 
registration is validated on an artificial testbed and on real data of a 
neuro clinical patient. 



1 Introduction 

The development of medical imaging in the last decades quickly triggered intense 
interest from the medical world to translate this progress on the imaging side to 
clinical diagnostics and treatment planning. In that respect image registration 
and in particular recently also the fusion of 2D and 3D image data is a fun- 
damental task in image-guided medical intervention. In [1 2D photographs of 
human faces are registered with a triangulated facial surface extracted from MRI 
data using rigid deformations. A registration method for sparse but highly accu- 
rate 3-D line measurements with a surface extracted from volumetric planning 
data based on the consistent registration idea and higher order regularization is 
introduced in [2]. 

The matching of photographic images with pre-operative MRI data is a par- 
ticular challenge in cranial neuronavigation. The photograph-MRI registration 
problem in the context of intracranial electroencephalography has been inves- 
tigated via a control point matching approach in [3 . Recently, normalized mu- 
tual information has been applied for the rigid transformation co-registration 
of brain photographs and MRI extracted cortical surfaces [4]. A major limita- 
tion of note, however, is that due to the brain shift the surgeon's view of the 
operating site is not in a rigid transformation correspondence to pre-operative 
images. Indeed, standard intracranial neuronavigation devices do not correct for 
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this movement of brain [5 . The main contributions of this paper are a novel clas- 
sification method for crease pattern such as sulci on implicit (cortical) surfaces 
and the actual 2D-3D registration method, where a non-rigid 3D deformation of 
the cortical surface is identified based on user marked sulci on photographs and 
the camera parameters. 



2 Materials and Methods 



The aim of this paper is to register a photograph of the exposed human cor- 
tex with the cortex geometry extracted from an MRI data set, see Fig. [l] us- 
ing the sulci as fiducials. The main ingredients of the proposed approach are a 
sulci classification on the cortex geometry (Section [2?T] ) and on the photograph 
(Section |2.2| ), as well as a model that uses the two classifications to register 
photograph and cortical ge- 



ometry (Section 2.3). The 
2D digital photographs were 
taken intra-operatively after 
supratentorial craniotomy and 
durotomy, and before cortico- 
tomy, using a digital camera 
with 10 mega pixel resolution 




Fig. 1. Input photograph, MRI graph surface 



positioned 20 cm above the craniotomy. The MR imaging was performed on a 3T 
MRI scanner. A Tl-weighted MP2RAGE sequence (1 x 1 x 1 , 256 x 256 x 176 
matrix) was segmented into gray and white matter using Brain Voyager QX [6] 
and then converted into a signed distance function of the cortical surface using 
a fast marching method. 



2.1 Sulci Classification on MRI Data 

In this section, we describe how to classify creases on the contour surface of 
a 3D object B C f2 represented via its signed distance function d : f2 ^ M 
on a computational domain f2. In the application the object is a brain volume 
and the creases are 
the sulci on the cor- 
tical surface. We aim 
for a moment based 
analysis of the (corti- 
cal) surface C := dB 
and define the zero Fig. 2. Moment-based classification 

moment shift of the 

implicit surface C as follows M^[B]{x) = |^ ^^^^^ d{y){y — x)dy which re- 

turns larger values in flat regions of C than in edge regions and even smaller near 
corners (cf. [7] for a related moment based classification on explicit surfaces). We 
define the scalar classification C{x) = (^\\M^[B]{x)\\/e'^) , where gf3{t) = j^^- 
Fig. [2] illustrates the behavior of the classifier C on three simple shapes and a 
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cortical surface extracted from an MRI using a white-green-blue-red color cod- 
ing. We observe a robust distinction for a single set of parameters (/3 = 20 and 
e = Sh or 4/i, where h denotes the grid width). 



2.2 Generation of Annotated Cortex Photographs 



Essential problems for the classification of 
sulci on photographs are additional struc- 
tures and their misinterpretation. Most 
prominent are cortical veins, which in 
addition partially occlude sulci (these 




veins are almost invisible in MRI in the ^.^ ^ Cortex photograph and die- 
used MP2RAGE sequence). On this back- ^-^^^^^ ^^^^ pre-classification 
ground we here confine to a still manual 

marking of sulci by an expert who is supported by the results of a prior auto- 
matic pre-classification of sulci based on learned discriminative dictionaries, cf. 
Fig. [sj where the method from [8 is used. 



2.3 Registration of Photograph and Cortex Geometry 



The co-registration of an MRI extracted cortical surface C C and a pho- 
tograph to compensate for effects such as the brain shift is based on a co- 
registration of the sulci classifiers on C and the photograph. To this end, we 
suppose C to be described as a graph 

C = {{x,z{x)) e R^\x e uj} 

with parameter domain C and graph 
function z : cu ^ We are interested in 
a local registration described by the cran- 
iotomy, where such a graph representation 
can be easily derived from the signed dis- 
tance function used in Section 12.11 Fur- 
thermore, let g G L'^{Q) denote the sulci 
classifie r on the photograph domain Q (cf. 
Section 2.2), and / G L'^{uj) the corresponding classifier on the cortical surface 
given as a function on the parameter domain cj, obtained by a suitable clamping 




Fig. 4. Geometric configuration 



of C and rescaling of the values to the unit interval [0, 1] (cf. Section 2.1). Both 
classifiers are supposed to be close to 1 in the central region of the sulci and small 
outside. Finally, we denote by P : ^ i? the projection of points in onto 
the image plane Q derived from known camera parameters. Let us remark that 
we thereby implicitly rule out self occlusions of the graph surface C under the 
image plane projections. Now, we ask for a deformation iZ^ : ^ defined on 
the parameter domain uj of the graph function z that matches C to its deformed 
configuration represented under the projection P in the photograph. Thereby, 
matching is encoded via the coincidence of the surface classifier f{x) on the MRI 
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described cortical surface and the image classifier g{P{ilj{x))) evaluated at the 
projected deformed position P(?/^(x)) for all x on the parameter domain uj. Thus, 
proper matching can be encoded via the minimization of the matching energy 

i?„.tcHM = I ljg{PWx))) - f{x)fA{x)dx 

based on a surface integral over C with the area element A{x) = (1 + |Vz(x)p)^, 
to consistently reflect the cortex geometry. In the overall variational approach 
the matching energy is complemented by a suitable elastic regularization energy, 
which acts as a prior on admissible deformations ip. Here, we consider the second 
order, elastic energy 

E..,m = \^Mx)f + + - Az{x))fdx. 

Note that a simple first order regularization like the Dirichlet energy of the dis- 
placement ip — (•, is not sufficient since matching information is mostly given 
on a low dimensional subset where proper nonlinear extrapolation is required 
and bending modes play an important role. Obviously, E,^^ is rigid body motion 
invariant (cf. [9]). Finally, we combine the matching energy £^match and the regu- 
larization energy ^^^eg to the total energy functional E[iIj] = ^^matchiV^] + ^E^^^lip] 
on deformations encoding the deformation of the cortical surface C, where A 
is a positive constant controlling the strength of the regularization. 

To minimize the objective functional we use a time discrete regularized gra- 
dient descent taking into account a suitable step size control combined with a 
cascadic descent approach to handle the registration in a coarse to fine manner. 
The first variation necessary for the descent algorithm is 

{^'M,C)= / [g{P{^{x)))-f{x)]Vg{P{4>{x)))-DP{4>{x))ax)A{x)dx 

J UJ 

+ \ [ Z\2^2, /i'(^3 - Z)) ■ Cdx, 

where the natural boundary conditions d^Aip = Aip = on 9a; for the nor- 
mal V on duj are considered and ip is initialized as the identity on C, i.e. 
ip{x) = {x^zix)). For the spatial discretization we consider bilinear Finite El- 
ements on a rectangular mesh overlaying uo and i7, and approximate the bi- 
Laplacian by the squared standard discrete Laplacian A\ = M~^LM~^L. 
Here, M and L denote the standard (lumped) mass and stiffness matrices, re- 
spectively. 

3 Results 

We have applied our registration approach both to test data and to real data. 
For the test data a cortical surface segmented on a 3D MRI data set has been 
taken as input together with an image generated from this surface via a given 
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manual 
segmentation 



moment-based 
segmentation 



before registration after registration 



Fig. 5. Both for the test data (top row) and the pair of a true photograph and 
an MRI extracted cortical surface (bottom row) we show (from left to right) the 
input image with marked sulci (computed image and real photograph respec- 
tively), the sulci classification on the 3D cortical surface segmented from MRI 
data, the initial misfit of the sulci marking on the 2D image projected on the 
cortical surface overlaying the cortical surface itself, and the final registration 
result. 

projection concatenated with an additional 3D nonrigid deformation. Then on 
the projected image selected sulci have been marked by hand. This pair of input 
data together with the computed registration result are depicted in Fig. |5j 

Furthermore, we considered a photograph of the brain surface seen through 
a left fronto-temporo-parietal craniotomy performed in a patient before place- 
ment of a subdural electrode grid for investigation of drug-resistant cryptogenic 
epilepsy. We segmented sulci using the procedure in Section [2^ This is then reg- 
istered via the proposed approach with the cortical surface extracted from the 
corresponding pre-interventional 3D MRI data. Again input data and achieved 
registration are shown in Fig. [5] 

4 Discussion 

We have proposed a novel method for the registration of photographs (2D) of the 
brain with the cortical surface extracted from 3D MRI data. The method turns 
out to be effective and robust both on test and on real data. It can be considered 
as an alternative to intra-operative MRI allowing subsequent co-registration with 
neuronavigation [10 . Currently, we aim for a validation study with an increased 
number of cases considering also data of patients with substantially smaller cran- 
iotomies. 

Furthermore, there is potential, that the iterative 2D/3D surface registration 
of digital images together with morphological 3D MRI data sets will enable to 
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build up a "dictionary" of brain surface features. Ultimately, the creation of such a 
dictionary might, to a certain extent, permit "intelligent" automatic recognition 
of brain surface features, where the 2D brain surface, seen through the intra- 
operative microscope standardly used during intra- cerebral procedures, would 
directly be co-registered to pre-interventional 3D MRI data. 

As already discussed, the sensitivity of MRI and photography is substantially 
different for different anatomic structures, e. g. veins are very prominent on im- 
ages yet not on the MRI modality used here. One could incorporate multiple 
MRI sequences and fuse vein sensitive images with the present images to im- 
prove the registration results. Finally, let us remark that one could also consider 
stereo photographs of the deformed surface to improve the methods performance. 
In that respect our approach can easily be adapted summing over copies of the 
matching energy. 
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